Cooling a data center

ABSTRACT

A data center cooling system includes a server rack frame assembly that includes a plurality of bays defined along a lengthwise dimension of the frame assembly, each bay including a volume defined at least in part by a specified height that is orthogonal to the lengthwise dimension and a specified width that is parallel to the lengthwise dimension and sized to at least partially enclose at least one server rack configured to support a plurality of data center computing devices; and at least one cooling unit sized for a bay of the plurality of bays of the server rack frame assembly and configured to circulate a heated airflow from an open back side of the at least one server rack, cool the heated air, and circulate a cooling airflow through an open front side of the at least one server rack.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of, and claims priority under 35U.S.C. § 120 to, U.S. patent application Ser. No. 16/150,842, filed onOct. 3, 2018, which is a continuation of, and claims priority under 35U.S.C. § 120 to, U.S. patent application Ser. No. 15/479,740, filed onApr. 5, 2017, now U.S. Pat. No. 10,123,461, the entire contents of whichare incorporated by reference herein.

TECHNICAL FIELD

The present disclosure relates to systems and methods for cooling a datacenter and, more particularly, cooling electronic devices in a datacenter.

BACKGROUND

Electronics, such as servers and networking devices, in a data centergenerate heat that, if not dissipated or removed, can cause problems,including failure of such devices. In some cases, determining a propersize for cooling equipment for removing the heat maybe difficult. Forexample, during a lifetime operation of a data center (e.g., decades),electronics platforms can go through multiple generations or refreshes.Therefore, correctly sizing the cooling equipment for any particularoperational time period, while also accounting for other operationaltime periods (e.g., in the future) of the data center may be difficult.

SUMMARY

In a general implementation, a data center cooling system includes aserver rack frame assembly that includes a plurality of bays definedalong a lengthwise dimension of the frame assembly, each bay including avolume defined at least in part by a specified height that is orthogonalto the lengthwise dimension and a specified width that is parallel tothe lengthwise dimension and sized to at least partially enclose atleast one server rack configured to support a plurality of data centercomputing devices; and at least one cooling unit sized for a bay of theplurality of bays of the server rack frame assembly and configured tocirculate a heated airflow from an open back side of the at least oneserver rack, cool the heated air, and circulate a cooling airflowthrough an open front side of the at least one server rack.

In an aspect combinable with the general implementation, the specifiedwidth and the specified height of the bay are based on a form factor ofthe at least one server rack.

In another aspect combinable with any of the previous aspects, at leastone perimeter dimension of the cooling unit is based on a form factor ofthe at least one server rack.

In another aspect combinable with any of the previous aspects, the atleast one server rack includes two server racks, each of the two serverracks providing between 5 kW and 60 kW of computing power.

In another aspect combinable with any of the previous aspects, the atleast one perimeter dimension includes a cooling unit width and acooling unit height.

In another aspect combinable with any of the previous aspects, thecooling unit includes a cooling module; and at least one fan positionedto circulate heated airflow through the cooling module.

In another aspect combinable with any of the previous aspects, thecooling module includes an air-to-liquid heat exchanger.

In another aspect combinable with any of the previous aspects, thecooling unit further includes a modulating valve fluidly coupled to atleast one of a fluid inlet or a fluid outlet of the air-to-liquid heatexchanger.

In another aspect combinable with any of the previous aspects, theplurality of bays are arranged in a first row of bays and a second rowof bays that is positioned vertically above the first row of bays.

In another aspect combinable with any of the previous aspects, theplurality of bays are further arranged in a third row of bays that ispositioned vertically above the second row of bays.

In another aspect combinable with any of the previous aspects, theserver rack frame assembly includes a first portion that defines thefirst row of bays; a second portion stacked on top of the first portionthat defines the second row of bays; and a third portion stacked on topof the second portion that defines the third row of bays.

In another general implementation, a method of cooling a data centerincludes circulating, with at least one cooling unit, a cooling airflowfrom a human-occupiable workspace of a data center through an open frontside of at least one server rack configured to support a plurality ofdata center computing devices, the at least one server rack positionedin one of the bays of a plurality of bays of a server rack frameassembly that are defined along a lengthwise dimension of the frameassembly, each bay including a volume defined at least in part by aspecified height that is orthogonal to the lengthwise dimension and aspecified width that is parallel to the lengthwise dimension and sizedto at least partially enclose the at least one server rack, the at leastone cooling unit sized for one of the bays of the plurality of bays ofthe server rack frame assembly; circulating, with the at least onecooling unit, a heated airflow that includes heat from the plurality ofdata center computing devices through an open back side of the at leastone server rack into a warm air aisle arranged adjacent the frameassembly; circulating, with the at least one cooling unit, the heatedairflow from the warm air aisle and through the at least one coolingunit to cool the heated airflow; and circulating the cooling airflowfrom the at least one cooling unit into the human occupiable workspace.

In an aspect combinable with the general implementation, the specifiedwidth and the specified height of the bay are based on a form factor ofthe at least one server rack, and the at least one perimeter dimensionof the cooling unit is based on the form factor of the at least oneserver rack.

Another aspect combinable with any of the previous aspects furtherincludes cooling the heated air with an air-to-liquid heat exchanger ofthe cooling unit; and controlling a flow of a cooling liquid to theair-to-liquid heat exchanger with a modulating valve that is fluidlycoupled to at least one of a fluid inlet or a fluid outlet of theair-to-liquid heat exchanger.

In another aspect combinable with any of the previous aspects, theplurality of bays are arranged in a first row of bays and a second rowof bays that is positioned vertically above the first row of bays.

Another aspect combinable with any of the previous aspects furtherincludes circulating the heated airflow from the at least one serverrack that is mounted in one of the bays in the first row of bays to theat least one cooling unit that is mounted in one of the bays in thesecond row of bays; and circulating the cooling airflow, with the atleast one cooling unit that is mounted in the one of the bays in thesecond row of bays, from the human-occupiable workspace through the atleast one server rack that is mounted in the one of the bays in thefirst row of bays.

Another aspect combinable with any of the previous aspects furtherincludes circulating the heated airflow from the at least one serverrack that is mounted in one of the bays in the first row of bays to theat least one cooling unit that is mounted in one of the bays in thesecond row of bays that is positioned vertically above the one of thebays in the first row of bays; and circulating the cooling airflow, withthe at least one cooling unit that is mounted in the one of the bays inthe first row of bays, from the human-occupiable workspace through theat least one server rack that is mounted in the one of the bays in thefirst row of bays.

Another aspect combinable with any of the previous aspects furtherincludes selecting the at least one cooling unit based at least in parton an expected heat output of the at least one server rack and a coolingcapacity of the at least one cooling unit.

In another aspect combinable with any of the previous aspects, the atleast one server rack includes a plurality of server racks.

Another aspect combinable with any of the previous aspects furtherincludes adjusting a power density of the plurality of server racks; andbased on the adjusted power density, adjusting a cooling capacity of theat least one cooling unit.

In another aspect combinable with any of the previous aspects, adjustinga power density of the plurality of server racks includes at least oneof adding a specified number of server racks to the plurality of serverracks; removing a specified number of server racks from the plurality ofserver racks; or replacing at least a portion of the plurality of serverracks with another portion of server racks that include an increased orreduced power density of the replaced portion of the plurality of serverracks.

In another aspect combinable with any of the previous aspects, the atleast one cooling unit includes a plurality of cooling units, andadjusting the cooling capacity of the at least one cooling unit includesat least one of adding a specified number of cooling units to theplurality of cooling units; removing a specified number of cooling unitsfrom the plurality of cooling units; or replacing at least a portion ofthe plurality of cooling units with another portion of cooling unitsthat include an increased or reduced cooling capacity of the replacedportion of the plurality of cooling units.

In another general implementation, a data center cooling system includesat least two rows of server racks positioned within a data centerbuilding. Each row includes a server rack frame assembly that includes aplurality of bays defined along a lengthwise dimension of the frameassembly, each bay including a volume defined at least in part by twospecified dimensions, where the plurality of bays include a firstportion of bays within the frame assembly along the lengthwise dimensionat a first level and a second portion of bays within the frame assemblyalong the lengthwise dimension at a second level above the first level;a plurality of server racks configured to support a plurality of datacenter computing devices, at least one server rack positioned in a bayin the first portion of bays and at least another server rack positionedin a bay in the second portion of bays; and at plurality of coolingunits, each sized for a bay of the plurality of bays of the server rackframe assembly, at least one cooling unit positioned in a bay in thefirst portion of bays and at least another cooling unit positioned in abay in the second portion of bays. The system further includes a controlsystem communicably coupled to the plurality of cooling units, thecontrol system configured to control the plurality of cooling units toperform operations including circulating a heated airflow from open backsides of the plurality of server racks through the plurality of coolingunits; cooling, in the plurality of cooling units, the heated air; andcirculating a cooling airflow through open front sides of the atplurality of server racks.

In an aspect combinable with the general implementation, the twospecified dimensions include a specified height that is orthogonal tothe lengthwise dimension and a specified width that is parallel to thelengthwise dimension.

In another aspect combinable with any of the previous aspects, the twospecified dimensions are based on a form factor of a particular serverrack of the plurality of server racks.

In another aspect combinable with any of the previous aspects, eachcooling unit includes at least one cooling coil fluidly coupled to acooling liquid junction of a plurality of cooling liquid junctions.

In another aspect combinable with any of the previous aspects, eachcooling liquid junction fluidly coupled to a source of a cooling liquidand to two or more cooling coils.

In another aspect combinable with any of the previous aspects, eachcooling unit further includes a modulating valve fluidly coupled to atleast one of a fluid inlet or a fluid outlet of the cooling coil.

In another aspect combinable with any of the previous aspects, theplurality of bays are further arranged in a third portion of bays thatis positioned vertically above the second portion of bays.

In another aspect combinable with any of the previous aspects, theserver rack frame assembly includes a first frame sub-assembly thatdefines the first portion of bays; a second frame sub-assembly stackedon top of the first portion that defines the second portion of bays; anda third frame sub-assembly stacked on top of the second portion thatdefines the third portion of bays.

Implementations according to the present disclosure may include one ormore of the following features. For example, implementations of a datacenter cooling system according to the present disclosure may providefor a match or substantial match of cooling capacity to heat load frominformation technology (IT) equipment in a data center. Also, the datacenter cooling system may, by matching or substantially matching coolingcapacity to heat load, operate at a highest efficiency. As anotherexample, electrical power and cost overhead may be reduced whileimproving power usage effectiveness (PUE) (i.e., a ratio of an amount ofelectrical power used to power IT equipment in a data center and totalamount of electrical power used by the data center). As a furtherexample, implementations of a data center cooling system according tothe present disclosure may include one or more cooling modules that neednot be preinstalled in an entirety of a data center building prior to ITequipment deployment and, instead, may be installed stepwise withdeployment (or redeployment) of the IT equipment. For example, coolingequipment may be deployed along with deployment of a zone (e.g., group,row, cluster, or otherwise) of IT equipment, such as server racks thathold multiple computing devices.

Implementations according to the present disclosure may also include oneor more of the following features. In some examples, “as needed”deployment of the cooling equipment (e.g., only when deployment orredeployment of IT equipment requires) may accelerate a data centerconstruction or data center start-up process, reducing the time to bringup services in a data center, which has substantial financial advantages(e.g., lower/shorter committed capital expense). “As needed” deploymentof cooling equipment may also enable flexibility through the life of thedata center to readjust cooling capacity with changes to IT equipment(e.g., server racks and other equipment) without impacting the operationand production loads. As yet another example, a data center coolingsystem according to the present disclosure may enable increasedflexibility and usage of a total volume of workspace (e.g.,human-occupiable, IT equipment occupiable), including volume above aconventional human work zone (e.g., up to 6-8 feet) of a data center. Asyet another example, a data center cooling system according to thepresent disclosure may allow for increased flexibility of an arrangementand rearrangement of IT equipment (e.g., server racks) and cooling unitsby standardizing a space required for both server racks and coolingunits. Further, a data center cooling system according to the presentdisclosure may facilitate additional cooling expansion that could enablepower-dense payload technologies relative to convention server rackpayloads. Also, a data center cooling system according to the presentdisclosure facilitates upgrades and specialized cooling modulesdepending on technical requirements of the payload.

The details of one or more implementations of the subject matterdescribed in this disclosure are set forth in the accompanying drawingsand the description below. Other features, aspects, and advantages ofthe subject matter will become apparent from the description, thedrawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1C are schematic illustrations of a side, top, and end view,respectively, of an example implementation of a data center system thatincludes a single, mixed computing and cooling layer according to thepresent disclosure.

FIGS. 2A-2C are schematic illustrations of a side, top, and end view,respectively, of an example implementation of a data center system thatincludes multiple, mixed computing and cooling layers according to thepresent disclosure.

FIGS. 3A-3C are schematic illustrations of an isometric, front, and sideview, respectively, of an example implementation of a cooling unit thatmay be used in either of the data center systems of FIGS. 1A-1C or FIGS.2A-2C.

FIG. 4 illustrates an example control loop for controlling a data centercooling unit according to the present disclosure.

FIG. 5 is a schematic illustration of an example controller for a datacenter cooling system according to the present disclosure.

DETAILED DESCRIPTION

Data center systems according to the present disclosure may comprise oneor more layers of mixed computing/cooling equipment. In some aspects,the layers comprise rows of server rack frame assembly structures thatsupport (e.g., enclose or hold) server racks that support computerserver devices. In the case of multiple layers, the layers may bestacked to form stacked rows of mixed computing/cooling equipment. Insome aspects, data center systems according to the present disclosuremay provide a scalable modular cooling scheme for a data center, whichcould be deployed (and redeployed, and rearranged) throughout the lifeof a data center.

In some aspects of data center systems described in the presentdisclosure, cooling units within the layer(s) of mixed computing/coolingequipment may comprise a form factor (e.g., external dimensions, overallshape, size) that is the same as or substantially similar to (e.g.,within inches or less of) a server rack that fits within the layer(s) ofmixed computing/cooling equipment. In some aspects, the similar oridentical “form factor” of the cooling unit may also include similarpower (e.g., voltage, wattage), mechanical considerations (e.g., weight,connections within the frame assembly), network (e.g., type ofcommunication interface), and telemetry (e.g., sensors, controlequipment) as a server rack, thereby making a particular cooling unitinterchangeable (all or substantially) within a particular space (e.g.,bay of a frame assembly) with a particular server rack. Thus, a coolingunit may be installed in any appropriate volume of the server rack frameassembly in place of a server rack.

In particular implementations, a cooling unit that operates within thelayer(s) of mixed computing/cooling equipment may use a cooling liquid(e.g., chilled water or chilled glycol from one or more chillers,condenser water or other evaporatively cooled liquid, or otherwise) froma data center building central plant. The cooling unit utilizes thecooling liquid in an air-to-liquid heat exchanger (e.g., liquid orrefrigerant cooling coil) to cool a flow of heated air from multipleserver racks. The cooling unit may include one or more fans that arecontrollably operated to circulate the heated airflow from the serverracks to the cooling unit, through the air-to-liquid heat exchanger tocool the heated airflow, and from the cooling unit (as a coolingairflow) back, to the server racks.

The layer(s) of mixed computing/cooling equipment may be deployed in adata center building in an incremental process. For example, an initialamount of IT power load (that generates a particular heat load) may bedetermined. For an initial deployment, the IT power load may be in asingle zone (e.g., row, cluster) or in or in multiple zones (e.g.,multiple rows, multiple clusters, multiple layers of rows or clusters).Based on the determined IT power load (and corresponding heat load),cooling capacity (e.g., per zone) may be determined, thereby determininga number of cooling units needed per zone (e.g., based on a maximumcooling capacity per cooling unit). IT equipment (e.g., server racks)may be deployed in the data center (e.g., within server rack frameassemblies arranged in one or more vertical layers) along with thedetermined number of cooling units (also within the server rack frameassemblies arranged in one or more vertical layers). For example, thecooling modules may be deployed interspersed in between server rackswithin the frame assembly, e.g., in particular locations (e.g., bays) toaccount for cooling requirements dictated by the server rack placements.

The initial deployment process described above can be iterative andrepeated for technology refreshes (e.g., replacement of particularserver racks with other, higher/lower power server racks) or newdeployments (e.g., addition of server racks). In each iteration, basedon the iteratively determined IT load (and corresponding determined heatload), a number and/or location of cooling units can be re-calculated.For example, if low powered server racks are being replaced with higher(relatively) powered server racks to increase a server rack powerdensity of the data center or zone of the data center, the coolingcapacity may be recalculated and cooling units can be added for theadditional cooling power. Similarly, cooling units could be removed whenserver rack power density is reduced in the data center or zone of thedata center. With incrementally deployed cooling units, cooling could be“as needed” and properly sized based on the deployed IT equipment ineach zone (e.g., single layer row, portion of a single layer row,multi-layer row, or portion of a multi-layer row). For example, for lowpower density server racks, fewer cooling units may be deployed within aparticular zone, while a larger number of cooling units may be neededfor high power density server racks within a particular zone.

Incremental, mixed computing/cooling equipment deployment can beimplemented in different data center equipment architectures. Forexample, although the present disclosure describes single andmulti-level row-based architectures, other, non-linear architectures(e.g., single or multi-level clusters) can also be implemented accordingto the present disclosure. Further, although layers of mixedcomputing/cooling equipment architectures are shown, the coolingequipment may be extracted from the layers of computing equipment andimplemented, e.g., as overhead cooling, underfloor cooling, end of rowcooling, conventional CRAC cooling, or otherwise.

FIGS. 1A-1C are schematic illustrations of a side, top, and end view,respectively, of an example implementation of a data center system 100that includes a single, mixed computing and cooling layer 111 positionedin a human-occupiable workspace 104 of a data center building 102. Asshown in this example implementation, the data center system 100includes a layer 111 of server racks 114 and cooling units 116 that arepositioned in a server rack frame assembly 110. In this example, thelayer 111 is arranged in a row 108 (e.g., linear or substantially lineararrangement of the server rack frame assembly 110). In alternativearrangements, the layer 111 may be arranged in a non-linear frameassembly, such as, for instance, a circular or partially circular frameassembly or otherwise.

The server rack frame assembly 110 comprises structural members (e.g.,metal or non-metal, such as composite) arranged and connected to formmultiple bays 112 in the frame assembly 110. Each bay 112, in thisexample implementation, defines a volume within the frame assembly 110and may be the same or similar in volume and perimeter dimensions (e.g.,height “H”, width “W”, and depth “D”) as the other bays 112 in the layer111 of the server rack assembly 110. In some aspects, H may be about 10feet, W may be about 10 feet, and D may be about 3 feet, as an exemplaryset of dimensions.

In some aspects, the perimeter dimensions of the bays 112 (and thus thevolumes defined by the bays 112) in the server rack frame assembly 110are based on or associated with dimensions of the server racks 114. Forexample, in the illustrated implementations, each bay 112 is sized(e.g., at least W and H) according to dimensions of a double server rack114 (i.e., two server racks of 42U in height, two server racks ofbetween 18-22U in height, or other sized server racks). In some aspects,the server racks 114 may be 54U in height, although server rack frameassembly 110 may handle a variation of rack sizes. For example, theracks 114 may be about 50 inches wide each.

Furthermore, different combinations of racks 114 can be used in theexample implementation. For an example, four racks 114, each 24 inchesin width, can be used within the server rack frame assembly 110. Theracks 114 can also vary in depth. For example, the front faces of theserver racks 114 may be flush with a front side 122 of the assembly 110,while a rear of the racks 114 may extend various depths beyond backsides 124 of the assembly 110.

As shown in FIG. 1A, the server racks 114 support electronic devices,such as processors, memory modules, networking switches, batterymodules, and other server rack computing components, both physically byproviding structure for the devices to be placed in and electrically byproviding electric power to the devices from a main source of power(e.g., through an inverter, a transformer, or both). Generally, eachillustrated server rack 114 may be one of a number of server rackswithin the data center building 102, which may include a server farm ora co-location facility that contains various rack mounted computersystems. Each server rack 114 may define multiple slots that arearranged in an orderly and repeating fashion within the server rack 114,and each slot is a space in the rack into which a corresponding serverrack sub-assembly 118 can be placed and removed. For example, a serverrack sub-assembly 118 can be supported on rails that project fromopposite sides of the rack 114, and which can define the position of theslots. Also, although multiple server rack sub-assemblies 118 areillustrated as mounted within the rack 114, there might be only a singleserver rack sub-assembly.

The slots, and the server rack sub-assemblies 114, can be oriented withthe illustrated horizontal arrangement (with respect to gravity) asshown in FIG. 1A. Alternatively, the slots, and the server racksub-assemblies 118, can be oriented vertically (with respect togravity). Where the slots are oriented horizontally, they may be stackedvertically in the rack 114, and where the slots are oriented vertically,they may be stacked horizontally in the rack 114.

Server rack 114, as part of a larger data center for instance, mayprovide data processing and storage capacity. In operation, a datacenter may be connected to a network, and may receive and respond tovarious requests from the network to retrieve, process, and/or storedata. In operation, for example, the server rack 114 typicallyfacilitates the communication of information over a network with userinterfaces generated by web browser applications of users who requestservices provided by applications running on computers in thedatacenter. For example, the server rack 114 may provide or help providea user who is using a web browser to access web sites on the Internet orthe World Wide Web.

The server rack sub-assembly 118 may be one of a variety of structuresthat can be mounted in a server rack 114. For example, in someimplementations, the server rack sub-assembly 118 may be a “tray” ortray assembly that can be slidably inserted into the server rack 114.The term “tray” is not limited to any particular arrangement, butinstead applies to motherboard or other relatively flat structuresappurtenant to a motherboard for supporting the motherboard in positionin a rack structure. In some implementations, the server racksub-assembly 118 may be a server chassis, or server container (e.g.,server box). In some implementations, the server rack sub-assembly 118may be a hard drive cage.

Each server rack sub-assembly 118 can include a frame or cage, a printedcircuit board, e.g., motherboard, supported on the frame, and one ormore electronic devices 120, e.g., a processor or memory, mounted on theprinted circuit board. The electronic devices 120 can include, forinstance, processors, memories, hard drives, network switches, or otherIT components. Other appurtenances, such as cooling devices, fans,uninterruptible power supplies (UPS) (e.g., battery modules) can bemounted to the server rack sub-assembly 118 (or otherwise to a rack114).

With respect specifically to FIG. 1A, the cooling units 116 arepositioned adjacent the server racks 114 in the bays 112. In someaspects, each cooling unit 116 may comprise a form factor (e.g.,dimensions such as width and height) that is similar to or the same as aform factor (e.g., dimensions such as width and height) of the serverracks 114. In some aspects, while a width and a height of the coolingunits 116 are the same as or similar to the width and height,respectively, of the server racks 114, a depth of the cooling unit(e.g., as shown in FIG. 1B), may be different (e.g., larger) than adepth of the server racks 114. Thus, in some aspects, each cooling unit116 may be positioned in any of the bays 112, and each server rack 114may be positioned in any of the bays 112, thereby making the coolingunits 116 and server racks 114 interchangeable within the server rackframe assembly 110.

In some aspects, the cooling units 116 may be fluidly coupled to asource of the cooling liquid, such as a chiller plant, one or moreevaporative cooling units (e.g., cooling towers), one or more condensingunits (e.g., in the case of direct expansion cooling), a natural sourceof cooling liquid (e.g., lake, ocean, river, or other natural body ofwater), or a combination thereof. In some aspects, the cooling units 116may be stand-alone refrigerant-based (DX) cooling units fluidly coupledto one or more condensing units located external to the data centerbuilding 102 (e.g., conventionally known as “CRAC” units).

As show in more detail in FIGS. 1B-1C, a warm air aisle 117 is definedbetween adjacent rows 108 of server rack frame assemblies 110, with openback sides 124 of the server rack frame assemblies 110 facing the warmair aisle 117. Although only two rows 108 and one warm air aisle areshown in FIGS. 1B and 1C, the data center building 102 may housemultiple rows 108 with warm air aisles 117 defined between pairs of rows108 of the server rack frame assemblies 110. In this exampleimplementation, the open back sides 124 of the frame assemblies 110allow for airflow with minimal or no pressure gradient between the backsof the server racks 114 (which are also open) that face the warm airaisle 117 and the warm air aisle 117 itself. Similarly, the server rackframe assemblies 110 have, in this example, open front sides 122 thatface the cool air aisles 121. The open front sides 122 of the frameassemblies 110 allow for airflow with minimal or no pressure gradientbetween the fronts of the server racks 114 (which are also open) thatface the cool air aisle 121 and the cool air aisle 121 itself. Thus, insome aspects, an air pressure at the fronts and backs of the serverracks 114, which are open to the aisles 121 and 117, respectively, aresubstantially equal or equal to an air pressure within the aisles 121and 117, respectively.

In operation, the cooling units 116 circulate a cooling airflow 128through the front sides 122 of the server racks 114 (e.g., that are opento the human-occupiable workspace 104. The cooling airflow 128 receivesheat from electronic devices 120 in the racks 114 and warms the airflow128 to a heated airflow 126 that enters the warm air aisle 117. Theheated airflow 126 is drawn back into the cooling units 116 (e.g., byfans in the units 116) and cooled through the one or more cooling coils(e.g., by a flow of the chilled liquid, condenser water, refrigerant, oran electrically-powered cooler such as a Peltier cooler). The cooledairflow 128 is circulated (e.g., by the fans) back into thehuman-occupiable workspace 104 adjacent the front sides 122 of theserver racks 114 and server rack frame assemblies 110.

FIGS. 2A-2C are schematic illustrations of a side, top, and end view,respectively, of an example implementation of a data center system 200that includes multiple, stacked and mixed computing and cooling layers213 a-213 c positioned in a human-occupiable workspace 204 of a datacenter building 202. As shown in this example implementation, the datacenter system 200 includes three layers 213 a-213 c of server racks 214(or server racks 215 or server racks 217) and cooling units 216 that arepositioned in a server rack frame assembly 210. In this example, thelayers 213 a-213 c are arranged in a row 208 (e.g., linear orsubstantially linear arrangement of the server rack frame assembly 210).In alternative arrangements, the layers 213 a-213 c may be arranged in anon-linear frame assembly, such as, for instance, a circular orpartially circular frame assembly or otherwise. Although three layers213 a-213 c are shown in this example, fewer (e.g., two) or more layersmay be implemented without departing from the scope of the presentdisclosure.

The server rack frame assembly 210 comprises structural members (e.g.,metal or non-metal, such as composite) arranged and connected to formmultiple bays 212 in the frame assembly 210. Each bay 212, in thisexample implementation, defines a volume within the frame assembly 210and may be the same or similar in volume and perimeter dimensions (e.g.,height “H” of 10 feet, width “W” of 10 feet, and depth “D” of 3 feet) asthe other bays 212 in the layer 213 of the server rack frame assembly210. In some aspects, the perimeter dimensions of the bays 212 (and thusthe volumes defined by the bays 212) in the server rack frame assembly210 are based on or associated with dimensions of the server racks 214(or 215 or 217). For example, in the illustrated implementations, eachbay 212 is sized (e.g., at least 10′ W and 10′ H) according todimensions of a double server rack 214 (or 215 or 217) (i.e., two serverracks of 42U in height, two server racks of between 18-22U in height, orother sized server racks). In some aspects, the server racks 214 (or 215or 217) may be 54U in height, although server rack frame assembly 110may handle a variation of rack sizes. For example, the racks 214 (or 215or 217) may be about 50 inches wide each.

Furthermore, different combinations of racks 214 (or 215 or 217) can beused in the example implementation. For an example, four racks 214 (or215 or 217), each 24 inches in width, can be used within the server rackframe assembly 210. The racks 214 (or 215 or 217) can also vary indepth. For example, the front faces of the server racks 214 (or 215 or217) may be flush with a front side of the assembly 210, while a rear ofthe racks 214 (or 215 or 217) may extend various depths beyond backsides of the assembly 210.

As shown in FIG. 2A, groups of bays 212 may be arranged to form columns209 (e.g., a vertical group of three stacked bays 212). The bays 212 maybe further grouped, in this example, in clusters 211 a-211 d of bays 212that include a multi-dimensional (e.g., vertical and horizontal)grouping of bays 212. In this example implementation, clusters 211 a-211d are shown as three by three (e.g., three columns 209 of bays 212 bythree layers 213 of bays 212) groupings of nine total bays 212. In someaspects, a column 209 or cluster 211 of bays 212 may be part of a power,cooling liquid, or network sharing architecture. For example, in someaspects, separate electrical power conductors may provide electricalpower to the server racks 214 (or 215 or 217) and cooling units 216 thatare positioned in separate columns 209 or clusters 211 of bays 212,e.g., to ensure that a failure of one power conductor only affects theracks 214 (or 215 or 217) and/or cooling units 216 in that particularcolumn 209 or cluster 211. Likewise, in some aspects, separate coolingliquid conduits may provide a cooling liquid to the cooling units 216that are positioned in separate columns 209 or clusters 211 of bays 212,e.g., to ensure that a failure of one cooling liquid conduit onlyaffects the cooling units 216 in that particular column 209 or cluster211. Further, in some aspects, separate networking connectors mayprovide data communication to the server racks 214 (or 215 or 217) thatare positioned in separate columns 209 or clusters 211 of bays 212,e.g., to ensure that a failure of one network connector only affects theracks 214 (or 215 or 217) in that particular column 209 or cluster 211.

As shown in FIG. 2A, the server racks 214 (or 215 or 217) supportelectronic devices, such as processors, memory modules, networkingswitches, battery modules, and other server rack computing components,both physically by providing structure for the devices to be placed inand electrically by providing electric power to the devices from a mainsource of power (e.g., through an inverter, a transformer, or both).Generally, each illustrated server rack 214 (or 215 or 217) may be oneof a number of server racks within the data center building 202, whichmay include a server farm or a co-location facility that containsvarious rack mounted computer systems. Each server rack 214 (or 215 or217) may define multiple slots that are arranged in an orderly andrepeating fashion within the server rack 214 (or 215 or 217), and eachslot is a space in the rack into which a corresponding server racksub-assembly 218 can be placed and removed. For example, a server racksub-assembly 218 can be supported on rails that project from oppositesides of the rack 214 (or 215 or 217), and which can define the positionof the slots. Also, although multiple server rack sub-assemblies 218 areillustrated as mounted within the rack 214 (or 215 or 217), there mightbe only a single server rack sub-assembly.

The slots, and the server rack sub-assemblies 214, can be oriented withthe illustrated horizontal arrangement (with respect to gravity) asshown in FIG. 2A. Alternatively, the slots, and the server racksub-assemblies 218, can be oriented vertically (with respect togravity). Where the slots are oriented horizontally, they may be stackedvertically in the rack 214 (or 215 or 217), and where the slots areoriented vertically, they may be stacked horizontally in the rack 214(or 215 or 217).

Server rack 214 (or 215 or 217), as part of a larger data center forinstance, may provide data processing and storage capacity. Inoperation, a data center may be connected to a network, and may receiveand respond to various requests from the network to retrieve, process,and/or store data. In operation, for example, the server rack 214 (or215 or 217) typically facilitates the communication of information overa network with user interfaces generated by web browser applications ofusers who request services provided by applications running on computersin the datacenter. For example, the server rack 214 (or 215 or 217) mayprovide or help provide a user who is using a web browser to access websites on the Internet or the World Wide Web.

The server rack sub-assembly 218 may be one of a variety of structuresthat can be mounted in a server rack 214 (or 215 or 217). For example,in some implementations, the server rack sub-assembly 218 may be a“tray” or tray assembly that can be slidably inserted into the serverrack 214 (or 215 or 217). The term “tray” is not limited to anyparticular arrangement, but instead applies to motherboard or otherrelatively flat structures appurtenant to a motherboard for supportingthe motherboard in position in a rack structure. In someimplementations, the server rack sub-assembly 218 may be a serverchassis, or server container (e.g., server box). In someimplementations, the server rack sub-assembly 218 may be a hard drivecage.

Each server rack sub-assembly 218 can include a frame or cage, a printedcircuit board, e.g., motherboard, supported on the frame, and one ormore electronic devices 220, e.g., a processor or memory, mounted on theprinted circuit board. The electronic devices 220 can include, forinstance, processors, memories, hard drives, network switches, or otherIT components. Other appurtenances, such as cooling devices, fans,uninterruptible power supplies (UPS) (e.g., battery modules) can bemounted to the server rack sub-assembly 218 (or otherwise to a rack 214(or 215 or 217)).

Server racks 214, server racks 215, and server racks 217, as shown, maybe physically similar or identical (e.g., similar or identical height,width, depth, weight), but may provide different computing power and,thus, different heat output. For example, server racks 214 may be 15 kWracks, with electronic devices 220 that generate about 15 kW of heateach. Server racks 215 may be 30 kW racks, with electronic devices 220that generate about 30 kW of heat each. Server racks 217 may be 60 kWracks, with electronic devices 220 that generate about 60 kW of heateach. Thus, in some aspects, a number and computing power of theparticular racks (e.g., server racks 214, 215, and/or 217) within aparticular layer 213 a-213 c, a particular column 209, or a particularcluster 211 a-211 d of bays 212 may dictate a number of cooling units216 that are positioned within the particular layer 213 a-213 c,particular column 209, or particular cluster 211 a-211 d of bays 212.For example, the greater number of server racks 217 relative to serverracks 214 within any particular layer, column, or cluster may dictate(e.g., due to cooling requirements for the heat generated by theelectronic devices 220 in the racks) a greater number of cooling units216 within the particular layer, column, or cluster. Conversely, thegreater number of server racks 214 relative to server racks 217 withinany particular layer, column, or cluster may dictate (e.g., due tocooling requirements for the heat generated by the electronic devices220 in the racks) a fewer number of cooling units 216 within theparticular layer, column, or cluster.

In some aspects, each particular layer, column, or cluster (or othersub-unit of the server rack frame assembly 210) may be reconfigured overan operating lifetime of the data center system 200. For example,reconfiguration may include moving particular server racks 214 (or 215or 217) from particular bays 212 to other bays 212. In suchreconfigurations, a number of cooling units 216 may not change, but suchcooling units 212 may also be moved from particular bays 212 to otherbays 212 (e.g., to account for movement of heat sources from onelocation to another location). Reconfigurations may also includereplacing server racks 214 with higher power server racks 215 or 217within particular bays 212. In such reconfigurations, a number ofcooling units 216 may increase (e.g., to account for additional heatgenerated by the higher power racks 215 or 217) within a particularlayer, column, or cluster. Reconfigurations may also include replacingserver racks 217 with lower power server racks 215 or 214 withinparticular bays 212. In such reconfigurations, a number of cooling units216 may decrease (e.g., to account for less heat generated by the lowerpower racks 215 or 214) within a particular layer, column, or cluster.Reconfigurations may also include removing server racks 214 (or 215 or217) from particular bays 212. In such reconfigurations, a number ofcooling units 216 may decrease (e.g., to account for less heatgenerated) within a particular layer, column, or cluster. In suchreconfigurations, for example, when a particular bay 212 does notinclude any server rack 214 (or 215 or 217) or cooling unit 216, ablank-off panel (e.g., sheet metal or otherwise) may be installed acrossa width and height of the bay 212 to prevent airflow from beingcirculated through the empty bay 212.

While FIG. 1B shows the clusters 211 a-211 d within a single row 208 ofthe server rack frame assembly 210, each cluster 211 may represent aparticular configuration at a particular operating time of the datacenter system 200 for illustrative purposes. For example, cluster 211 amay represent a configuration at an initial operating time of the system200, with only four server racks 214 and one cooling unit 216 within thenine total bays 212 of the cluster 211 a. At the initial operating time,only a single cooling unit 216 may be needed to remove the heatgenerated by the four server racks 214. Empty bays 212 may includeblank-off panels as previously described.

Cluster 211 b may represent a configuration at a later operating time ofthe system 200, with eight server racks 214 and one cooling unit 216within the nine total bays 212 of the cluster 211 b. At the lateroperating time, still only a single cooling unit 216 may be needed toremove the heat generated by the eight server racks 214. In thisconfiguration, no bays 212 are empty.

Cluster 211 c may represent a configuration at another later operatingtime of the system 200, with seven server racks 215 and two coolingunits 216 within the nine total bays 212 of the cluster 211 c. At thislater operating time, two single cooling units 216 may be needed toremove the heat generated by the seven server racks 215, which, asdescribed, may have more computing power and thus generate more heatthan the server racks 214, and thus require additional cooling power(through the additional cooling units 216) to remove the generated heat.

Cluster 211 d may represent a configuration at still another lateroperating time of the system 200, with six server racks 217 and threecooling units 216 within the nine total bays 212 of the cluster 211 d.At this later operating time, three single cooling units 216 may beneeded to remove the heat generated by the six server racks 217, which,as described, may have more computing power and thus generate more heatthan the server racks 214 and 217, and thus require additional coolingpower (through the additional cooling units 216) to remove the generatedheat.

With respect specifically to FIG. 2A, the cooling units 216 arepositioned adjacent the server racks 214 (or 215 or 217) in the bays212. In some aspects, each cooling unit 216 may comprise a form factor(e.g., dimensions such as width and height) that is similar to or thesame as a form factor (e.g., dimensions such as width and height) of theserver racks 214 (or 215 or 217). In some aspects, while a width and aheight of the cooling units 216 are the same as or similar to the widthand height, respectively, of the server racks 214 (or 215 or 217), adepth of the cooling unit (e.g., as shown in FIG. 2B), may be different(e.g., larger) than a depth of the server racks 214 (or 215 or 217).Thus, in some aspects, each cooling unit 216 may be positioned in any ofthe bays 212, and each server rack 214 (or 215 or 217) may be positionedin any of the bays 212, thereby making the cooling units 216 and serverracks 214 (or 215 or 217) interchangeable within the server rack frameassembly 210.

In some aspects, the cooling units 216 may be fluidly coupled to asource of the cooling liquid, such as a chiller plant, one or moreevaporative cooling units (e.g., cooling towers), one or more condensingunits (e.g., in the case of direct expansion cooling), a natural sourceof cooling liquid (e.g., lake, ocean, river, or other natural body ofwater), or a combination thereof. In some aspects, the cooling units 216may be stand-alone refrigerant-based (DX) cooling units fluidly coupledto one or more condensing units located external to the data centerbuilding 202 (e.g., conventionally known as “CRAC” units). For instance,as shown in FIG. 2B, a rack structure 250 (e.g., pipe rack or otherwise)may be positioned to support cooling fluid supply 252 and cooling fluidreturn 254 conduits that are fluidly coupled to the source of coolingliquid as well as cooling fluid junctions 256. The rack structure 250may also support, in this example, power conductors 262 that provideelectrical power from a source of electrical power to the racks in theserver rack frame assemblies 210.

In this example, the cooling fluid junctions 256 may be mounted on ornear the server rack frame assemblies 210 and fluidly connected to,e.g., cooling coils, that are part of the cooling units 216. In someaspects, a single cooling fluid junction 256 may be fluidly coupled tosupply and return connections of cooling coils in several cooling units,e.g., all of the cooling units 216 in a particular layer, all of thecooling units 216 in a particular column, all of the cooling units 216in a particular cluster, or otherwise.

As show in more detail in FIGS. 2B-2C, a warm air aisle 219 is definedbetween adjacent rows 208 of server rack frame assemblies 210, with openback sides 224 of the server rack frame assemblies 210 facing the warmair aisle 219. Although only two rows 208 and one warm air aisle areshown in FIGS. 2B and 2C, the data center building 202 may housemultiple rows 208 with warm air aisles 219 defined between pairs of rows208 of the server rack frame assemblies 210. In this exampleimplementation, the open back sides 224 of the frame assemblies 210allow for airflow with minimal or no pressure gradient between the backsof the server racks 214 (or 215 or 217) (which are also open) that facethe warm air aisle 219 and the warm air aisle 219 itself. Similarly, theserver rack frame assemblies 210 have, in this example, open front sides222 that face the cool air aisles 221. The open front sides 222 of theframe assemblies 210 allow for airflow with minimal or no pressuregradient between the fronts of the server racks 214 (or 215 or 217)(which are also open) that face the cool air aisle 221 and the cool airaisle 221 itself. Thus, in some aspects, an air pressure at the frontsand backs of the server racks 214 (or 215 or 217), which are open to theaisles 221 and 219, respectively, are substantially equal or equal to anair pressure within the aisles 221 and 219, respectively.

In operation, the cooling units 216 circulate a cooling airflow 228through the front sides 222 of the server racks 214 (or 215 or 217)(e.g., that are open to the human-occupiable workspace 204. The coolingairflow 228 receives heat from electronic devices 220 in the racks 214(or 215 or 217) and warms the airflow 228 to a heated airflow 226 thatenters the warm air aisle 219. The heated airflow 226 is drawn back intothe cooling units 216 (e.g., by fans in the units 216) and cooledthrough the one or more cooling coils (e.g., by a flow of the chilledliquid, condenser water, refrigerant, or an electrically-powered coolersuch as a Peltier cooler). The cooled airflow 228 is circulated (e.g.,by the fans) back into the human-occupiable workspace 204 adjacent thefront sides 222 of the server racks 214 (or 215 or 217) and server rackframe assemblies 210.

FIGS. 3A-3C are schematic illustrations of an isometric, front, and sideview, respectively, of an example implementation of a cooling unit 300that may be used in either of the data center systems 100 or 200 ofFIGS. 1A-1C or FIGS. 2A-2C, respectively, as one or more of the coolingunits 116 or cooling units 216.

Cooling unit 300 includes a base 308 onto which are mounted an intakeplenum 302, a cooling coil 304, and a fan wall 307 that includes, inthis example, multiple (e.g., four) fans 306. The base 308, as shown maybe “palletized” in that it includes multiple slots 318 at a front sideof the base 308 for forklift forks (or other mechanized lifting device)to move the cooling unit 300. Thus, the cooling unit 300 may be movedand lifted, e.g., into bays of a server rack frame assembly, as a singlepiece unit. In some aspects, a similar palletized base can be installedonto one or more server racks (e.g., racks 114, 214, 215, 217), suchthat a common equipment mover (e.g., fork lift or an automated mover)can move and transport both the server racks and cooling units in a datacenter.

The intake plenum 302, in this example implementation, includes a seal310 that extends on lateral sides of the cooling unit 300 (and can alsoextend on top and bottom sides of the unit 300) to attach to, e.g., aserver rack frame assembly when the cooling unit 300 is mounted in abay. The seal 310 may also block airflow from circulating external tothe cooling coil 304, such as around the sides of the coil 304. In someaspects, the intake plenum 302 may also include a filtration device(e.g., media, UV lighting, or otherwise) to clear an airflow thatcirculates through the cooling unit 300.

The cooling coil 304 includes an inlet 314 that is connected to a supplyheader 322 and an outlet 312 (that can also be connected to a returnheader, not shown). In this example, the cooling coil 304 is anair-to-liquid heat exchanger that can receive a cooling liquid (e.g.,chilled water or glycol, condenser water, or other liquid which has atemperature less than a temperature of a heated airflow that enters thecoil 304) and cool a heated airflow to produce a cooling airflow. Amodulating control valve 316 is positioned in fluid communication withthe inlet 314 in this example, but also may be positioned in fluidcommunication with the outlet 312, to control a flow rate of the coolingliquid circulated to the cooling coil 304.

In this example implementation, the fans 306 are positioned on the fanwall 307 to draw an airflow through the cooling coil 304 and expel thecooled airflow from the cooling unit 300. Motor controllers 320 aremounted to the fan wall 307 in this example implementation andelectrically coupled to power and control the fans 306. The motorcontrollers 320 may be, for example, variable frequency drives, motorstarters for on/off control of the fans 306, or other type of motorcontroller. Although there are two motor controllers 320 shown in thisexample, there may be a 1:1 ratio of controllers 320 to fans 306 oreven, for this example, a 1:4 ratio of controllers 320 to fans 306 (orother ratio).

As shown in FIG. 3C, a controller 330 is communicably coupled to themotor controllers 320. The controller 330 may implement one or morecontrol schemes or loops to control the fans 306, e.g., to meet apredetermined pressure differential between a warm air aisle and a coolair aisle. In some aspects, the controller 330 may also be communicablycoupled to the control valve 316 to control (e.g., modulate to open orto close) a flow of the cooling liquid to the cooling coil 304, e.g., tomeet a leaving airflow temperature setpoint or an approach temperature(e.g., leaving air temperature minus entering cooling liquidtemperature) of the coil 304.

FIG. 4 illustrates an example control loop 400 for controlling a datacenter cooling apparatus 412. In some embodiments, the cooling apparatus412 may be similar to, for example, the cooling unit 300 shown in FIGS.3A-3C, or other cooling apparatus described in the present disclosure.In some embodiments, the control loop 400 may control the coolingapparatus 412 to maintain and/or adjust a flow of cooling fluid to thecooling apparatus 412 to meet an approach temperature set point of thecooling apparatus 412. The approach temperature set point, in someembodiments, may be a difference between a leaving air temperature fromthe cooling apparatus 412 and an entering cooling fluid temperature tothe cooling apparatus 412. In some embodiments, the illustrated coolingapparatus 412 may represent multiple cooling units in the data center,such as, for example, a group of cooling units, several groups ofcooling units, or all of the cooling units in the data center.

As illustrated, the control loop 400 includes an input value 402 and afeedback value 414 that are provided to a summing function 404. In someembodiments, the input value 402 may represent a desired valve position(e.g., percentage open value) of a particular valve (or group of valves)associated with a particular cooling apparatus 412 in the data center.For example, the desired valve position may be 100% open. In someembodiments, the desired valve position may be about 95% open, therebyproviding for some headroom for control of the valve.

The feedback value 414, in the illustrated embodiment, may represent thevalve position (i.e., percent open) of the particular valve associatedwith the cooling apparatus 412. For example, the particular valve may be“X” percentile valve as determined based on flow rate data in the datacenter. In some embodiments, the particular valve may be the 2^(nd)percentile control valve, meaning that about 98% of the valvesassociated with cooling apparatus in the data center are at, or are lessthan, the valve position (i.e., percent open) of the 2^(nd) percentilevalve.

The summing function 404 compares the input value 402 and the feedbackvalue 414 and provides an output value 406 to a controller 408. Ofcourse, in some embodiments, the summing function 404 is implemented bythe controller 408. In some embodiments, the controller 408 may be amain controller of the cooling system of the data center, which iscommunicably coupled to each control valve of the data center and/or theindividual controllers associated with the control valves. For example,the main controller may be a master controller communicably coupled toslave controllers at the respective control valves. In some embodiments,the controller 408 may be a Proportional-Integral-Derivative (PID)controller. Alternatively, other control schemes, such as PI orotherwise, may be utilized. As another example, the control scheme maybe implemented by a controller utilizing a state space scheme (e.g., atime-domain control scheme) representing a mathematical model of aphysical system as a set of input, output and state variables related byfirst-order differential equations.

The controller 408 receives and/or determines the output value 406(e.g., a difference between the input value 402 and the feedback value414). Based on the output value 406, the controller 408 may adjust anapproach temperature set point 410 communicated to the cooling apparatus412. For example, if the output value 406 defines a difference betweenthe input value 402 and the feedback value 414 greater than a thresholdvalue (e.g., more than 1° C.), the approach temperature set point 410may be adjusted by the controller 408.

If the feedback value 414 is less than the input value 402 (i.e., the Xpercentile valve is less open than desired), then the controller 408 mayadjust the approach temperature set point 410 downward (i.e.,decreased), thereby causing the control valve or valves to open andcirculate more cooling fluid to the cooling apparatus 412.Alternatively, if the feedback value 414 is more than the input value402 (i.e., the X percentile valve is more open than desired), then thecontroller 408 may adjust the approach temperature set point 410 upward(i.e., increased), thereby causing the control valve or valves to closeand circulate less cooling fluid to the cooling apparatus 412.

FIG. 5 is a schematic illustration of an example controller 500 (orcontrol system) for a data center cooling system according to thepresent disclosure. For example, the controller 500 may be communicablycoupled with, or as a part of, a data center control system thatincludes one or more cooling units, such as the cooling unit 300, tocontrol one or more fans, one or more control valves, or both fan(s) andvalve(s) of the module 300. In some aspects, the controller 500 mayinclude the controller 330 shown in FIG. 3C.

The controller 500 is intended to include various forms of digitalcomputers, such as printed circuit boards (PCB), processors, digitalcircuitry, or otherwise that is part of a vehicle. Additionally thesystem can include portable storage media, such as, Universal Serial Bus(USB) flash drives. For example, the USB flash drives may storeoperating systems and other applications. The USB flash drives caninclude input/output components, such as a wireless transmitter or USBconnector that may be inserted into a USB port of another computingdevice.

The controller 500 includes a processor 510, a memory 520, a storagedevice 530, and an input/output device 540. Each of the components 510,520, 530, and 540 are interconnected using a system bus 550. Theprocessor 510 is capable of processing instructions for execution withinthe controller 500. The processor may be designed using any of a numberof architectures. For example, the processor 510 may be a CISC (ComplexInstruction Set Computers) processor, a RISC (Reduced Instruction SetComputer) processor, or a MISC (Minimal Instruction Set Computer)processor.

In one implementation, the processor 510 is a single-threaded processor.In another implementation, the processor 510 is a multi-threadedprocessor. The processor 510 is capable of processing instructionsstored in the memory 520 or on the storage device 530 to displaygraphical information for a user interface on the input/output device540.

The memory 520 stores information within the controller 500. In oneimplementation, the memory 520 is a computer-readable medium. In oneimplementation, the memory 520 is a volatile memory unit. In anotherimplementation, the memory 520 is a non-volatile memory unit.

The storage device 530 is capable of providing mass storage for thecontroller 500. In one implementation, the storage device 530 is acomputer-readable medium. In various different implementations, thestorage device 530 may be a floppy disk device, a hard disk device, anoptical disk device, or a tape device.

The input/output device 540 provides input/output operations for thecontroller 500. In one implementation, the input/output device 540includes a keyboard and/or pointing device. In another implementation,the input/output device 540 includes a display unit for displayinggraphical user interfaces.

The features described can be implemented in digital electroniccircuitry, or in computer hardware, firmware, software, or incombinations of them. The apparatus can be implemented in a computerprogram product tangibly embodied in an information carrier, forexample, in a machine-readable storage device for execution by aprogrammable processor; and method steps can be performed by aprogrammable processor executing a program of instructions to performfunctions of the described implementations by operating on input dataand generating output. The described features can be implementedadvantageously in one or more computer programs that are executable on aprogrammable system including at least one programmable processorcoupled to receive data and instructions from, and to transmit data andinstructions to, a data storage system, at least one input device, andat least one output device. A computer program is a set of instructionsthat can be used, directly or indirectly, in a computer to perform acertain activity or bring about a certain result. A computer program canbe written in any form of programming language, including compiled orinterpreted languages, and it can be deployed in any form, including asa stand-alone program or as a module, component, subroutine, or otherunit suitable for use in a computing environment.

Suitable processors for the execution of a program of instructionsinclude, by way of example, both general and special purposemicroprocessors, and the sole processor or one of multiple processors ofany kind of computer. Generally, a processor will receive instructionsand data from a read-only memory or a random access memory or both. Theessential elements of a computer are a processor for executinginstructions and one or more memories for storing instructions and data.Generally, a computer will also include, or be operatively coupled tocommunicate with, one or more mass storage devices for storing datafiles; such devices include magnetic disks, such as internal hard disksand removable disks; magneto-optical disks; and optical disks. Storagedevices suitable for tangibly embodying computer program instructionsand data include all forms of non-volatile memory, including by way ofexample semiconductor memory devices, such as EPROM, EEPROM, and flashmemory devices; magnetic disks such as internal hard disks and removabledisks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Theprocessor and the memory can be supplemented by, or incorporated in,ASICs (application-specific integrated circuits).

To provide for interaction with a user, the features can be implementedon a computer having a display device such as a CRT (cathode ray tube)or LCD (liquid crystal display) monitor for displaying information tothe user and a keyboard and a pointing device such as a mouse or atrackball by which the user can provide input to the computer.Additionally, such activities can be implemented via touchscreenflat-panel displays and other appropriate mechanisms.

The features can be implemented in a control system that includes aback-end component, such as a data server, or that includes a middlewarecomponent, such as an application server or an Internet server, or thatincludes a front-end component, such as a client computer having agraphical user interface or an Internet browser, or any combination ofthem. The components of the system can be connected by any form ormedium of digital data communication such as a communication network.Examples of communication networks include a local area network (“LAN”),a wide area network (“WAN”), peer-to-peer networks (having ad-hoc orstatic members), grid computing infrastructures, and the Internet.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as descriptions offeatures specific to particular implementations of particularinventions. Certain features that are described in this specification inthe context of separate implementations can also be implemented incombination in a single implementation. Conversely, various featuresthat are described in the context of a single implementation can also beimplemented in multiple implementations separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. For example, exampleoperations, methods, or processes described herein may include moresteps or fewer steps than those described. Further, the steps in suchexample operations, methods, or processes may be performed in differentsuccessions than that described or illustrated in the figures.Accordingly, other implementations are within the scope of the followingclaims.

What is claimed is:
 1. A data center cooling system, comprising: aserver rack frame assembly that comprises a plurality of bays definedalong a lengthwise dimension of the frame assembly, each bay comprisinga volume defined at least in part by a specified height that isorthogonal to the lengthwise dimension and a specified width that isparallel to the lengthwise dimension and sized to at least partiallyenclose at least one server rack configured to support a plurality ofdata center computing devices; and at least one cooling unit sized for abay of the plurality of bays of the server rack frame assembly andconfigured to circulate a heated airflow from an open back side of theat least one server rack, cool the heated air, and circulate a coolingairflow through an open front side of the at least one server rack.